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Claims • 

I . A type of object-detecting device characterized by the fact that it has the following 

means: 

an image-pickup means that captures the image from the front of the vehicle itself 
an image judgment means that performs image processing for the image captured with 
said image-pickup means (hereinafter to be referred to as the "picked-up image") and judges 
whether said picked-up image was captured when the vehicle itself was balanced. 
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and an object position computing means that performs the following operation: when it is 
judged by said image judgment means that said picked-up image was captured when the vehicle 
itself was not balanced, the position of the object in said picked-up image is determined based on 
the picked-up image capuired when the vehicle itself was balanced. 

2. The object-detecting device described in Claim 1 characterized by the following facts: 
it also has an acceleration-computing means that computes the acceleration of the object 

present in said picked-up image (hereinafter to be referred to as 'Image acceleration''); 

said image judgment means judges that said picked-up image corresponding to zero 
image acceleration as computed by said acceleration computing means is an image captured 
when the vehicle itself was balanced. 

3. The object-detecting device described in Claim 2 clmacterized by the following focts: 
it also has a velocity computing means that performs image processing for the picked-up 

image captured with said image pickup means and computes tlie velocity of the object present in 
said picked-up image in the vertical direction of the image (hereinafter to be referred to as 
'*image velocity''); 

said image judgment means judges that said picked-up image, which has zero image 
acceleration as computed by said acceleration computing means, and \\h'\ch has positive or 
negative image velocity as computed by said velocity computing means is an image captured 
when the vehicle itself was balanced. 

4. The object-detecting device described in any of Claims 1-3 characterized by the fact 

that 

it also has an object information computing means that performs the following operation: 
when it is judged by said image judgment means that said picked-up image was captured when 
the vehicle itself was not balanced, the size of the object present in said picked-up image is 
computed based on the picked-up image captured when the vehicle itself was balanced, and 
based on the computing result, the distance from said image pickup means to the object present 
in said picked-up image is computed. 

5. The object-detecting device described in Claim 4 characterized by fact that 

said object position computing means performs the following operation: when it is judged 
by said image judgment means that said picked-up image was captured when the vehicle itself 
was not balanced, based on the distance from said image pickup means to the object present in 
said picked'Up image computed by means of said object information computing means, the 
vision axis of said image pickup means at the time when said picked-up image was captured is 
computed, and based on the computed vision axis of said image pickup means, the position of 
the object in the image is computed. 

6. An object-detecting method characterized by the following facts: 
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the image from the front of the vehicle itself and captured by an image pickup means 
(hereinafter to be referred to as the "picked-up image'') is subjected to image processing, and a 
judgment is made as to whether said picked-up image was captured when the vehicle itself was 
balanced; 

when it is judged that said picked-up image was captured when the vehicle itself was not 
balanced, the position of the object present in said picked-up image is computed based on the 
picked-up image captured when the vehicle itself was balanced. 

I Document name] Specification 

Title of the invention: Object-detecting device and object-detecting method 

Technical field 

[0001] 

The present invention pertains to an object-detecting device and an object-detecting 
method for detecting the presence of an object the front of the vehicle itself 

Background technology 
[0002] 

Patent Reference 1 disclosed a type of position-detecting device characterized by the 
following facts: plural objects are extracted from the images captured by an onboard camera; 
based on the variation over time of the position in y-direction (height) of said plural objects, 
correction of the y-coordinates of the objects is performed taking into consideration the pitching 
and other behavior of the vehicle. As a result, it is possible to detect the position of each object 
by excluding the influence of the pitching and other behavior of the vehicle. 

[0003] 

[Patent Reference 1 1 Japanese Kokai Patent Application No. 2001-84497 

Disclosure of the invention 

Problems to be solved by the invention 

[0004J 

However, because detection of the y-coordinate is per formed based on the variation over 
time of the position of the object in y-direction, it is impossible to detect variation in the position 
of the object due to pitching Uiat often talces place due to passejigers, cargo, etc. This is 
undesirable. 
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Means for solving the problems 
[0005] 

This invention is characterized by the following facts: the image from the front of the 
vehicle itself and captured with an image pickup means (hereinafter to be referred to as the 
"picked-up image'') is subjected to image processing, and a judgment is made as to whether said 
pickcd-up image was captured when the vehicle itself was balanced; when it is judged that said 
picked-up image was captured when the vehicle itself was not balanced, the position of the 
object in said picked-up image is computed based on the picked-up image captured v^hen the 
vehicle itself was balanced. 

Effects of the hivention 
[0006] 

According to the present invention, a judgment is made as to whether the picked-up 
image was captured when the vehicle itself was balanced; if it is judged that the picked-up image 
was captured when the vehicle itself was not balanced, the position of the object present in said 
picked-up image is computed based on the information about the object computed from the 
picked-up image captured when the vehicle itself was balanced. As a result, even when pitching 
often takes place due to passengers or cargo, it is determined that said picked-up image was 
captured when the vehicle itself was not balanced, and it is possible to compute the position of 
the object in the image correctly. 

Optimum embodiment of the invention 
[0007] 

Figure 1 is a block diagram illustrating an example of the constitution of an embodiment 
of the object-detecting device in tlie present embodiment. Said object-detecting device (100) is 
mounted on board the vehicle, and it includes camera (1), image memory (2), microcomputer (3), 
and display unit (4). Said camera { I ) is arranged at the front of the vehicle, and it takes pictures 
at a constajit time interval At. Said image memory (2) converts the image captured with camera 
(I) to digital data and stores it. Said microcomputer (3) leads the digital image stored in image 
memory (2). As will be explained later, the size of the object in real space and the distance to the 
object are detected, taking into consideration the pitching of the vehicle. Said display unit (4) 
displays the detected object in a bird's-eye-view mapped downward onto a map with the vehicle 
itself at the center. 
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[0008] 

Also, in the present application example, it is assumed that in the image captured with 
camera ( I ), only one moving object, such as another vehicle, is present, and the real-space size o f 
the object present in the image has a v^'idth tliat can be detected on the image. 

[0009] 

Figure 2 shows an example of change in the position of the object present the front of the 
vehicle in the picked-up image when pitching of the vehicle takes place. As shown in Figure 2(a), 
assuming the deviation angle to be 0o when the object is viewed from the camera vision axis 0 
with respect to the hoi*izontal direction, y-coordinate value yo in the picked-up image when the 
vehicle is balanced (to be defined later) is computed using the following Equation (1): 

yo^aGo- * ' ( 1 ) 

Here, a is a constant that can be uniquely computed from the image pickup element size, 
focal distance, etc., of camera (1 ) (hereinafter to be referred to as the ''camera parameters''). 

[0010] 

hi this case, when pitching of the vehicle takes place, and the pi tching angle in this case 
is 0p, y-coordinate yo' of the object in die image is computed using the following Equation (2): 

y o' = aeo' = a ( 9o-^0p ) ^ y o +aep' • - ( 2 ) 

That iSj the change in the y-coordinate of the object in the image when pitching takes 
place can be seen to be proportional to the size of pitching angle 0p. Consequently, Figure 3 
shows the relationship of pitching angle 0p and the y-axis acceleration of the object in the image, 
that is, the vertical acceleration of the image (image acceleration). 

[0011] 

As showji in Figure 3, characteristic curve (3a) shows the up/down periodic movement 
(periodic movement (ja)) due to pitching when pitching takes place; characteristic curve (36) 
shows the change in image velocity over time (image velocity (3b)); and characteristic curve (3c) 
shows the change in image acceleration (image acceleration (3c)) over time. As shown in Figure 
3, when image acceleration (3c) is zero, image velocity (3b) is maximum or minimum, and 
periodic movement (3a) is at the inflection point. Because the inflection point of periodic 
movement (3a) shows the point at wJiich the vehicle is balanced, it is possible to judge that an 
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image with an image acceleration (3c) of zero is one captured when the vehicle itself was 
balanced when pitching of the vehicle itself took place, 

[0012] 

Also, in this embodiment, edge extraction processing for the picked-up image allows the 
well-known gradient method and block matching method to be adopted to compute the optical 
How, and the velocity of the object present in the image is detected. As a result, said image 
velocity (3b) and image acceleration (3c) are detected. The edge extraction processing and the 
optical flow computation processing are well-known technologies, and will not be explained 
again. 

[0013] 

Based on the image for which image acceleration (3c) in the y-direction is zero for the 
object detected on the image, that is, the image captured when the vehicle itself was balanced 
(balanced image), it is possible to compute the y-coordinate of the object in the image, to be 
explained later, the width of the object in real space (the size of the object), and the distance from 
camera (1) to the object, that is, distance D from the focal position of camera (1) to the object. 
Figure 4 shows an example of the situation in which the image balanced is used to compute the 
y-coordinate of the object in the image, the width of the object in real space, and the distance 
between the focal position of camera (1) and the object. Figure 4(a) shows a side view of the 
object, and Figure 4(b) shows aji top view of the object. 

[0014] 

As shown in Figure 4(a), using vision axis 0 of camera (1 ) and apparent angle of the 
object 00 that can be detected in the image, the y-coordinate of the object in the image can be 
computed using the following Equation (3), 

y =a ( 0 + 90 ) • • • ( 3 ) 

Also, assuming the pre-measurcd camera mounting height to be H, the vision axis of 
camera (1) to be 0, and the apparent angle of the object to be Oo. distance D from the focal 
position of camera (1 ) to liie object is D can be computed using the following Equation (4). 



D = H / t a n ( OH-Oo ) • • • { 4 ) 
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[0015] 

Width Ws of the object is then computed. As shown hi Figure 4(b), based on the 
relationship betweeji image width xw and lateral angle 0x of the object, the following Equation 
(5) is obtained. 

X w = 3ex' ' • ( 5 ) 

Here, p is a constant that can be uniquely computed from the camera parameters. 

[0016] 

Consequently, using Equation (5), one can compute width Ws of the object using the 
following Equation (6). 

Ws =ex 'D=xw/(i'D— -(6) 

Consequently, based on the image balanced, it is possible to compute the y-coordinate of 
the object, width Ws of the object, and distance D from the focal position of camera (1) to the 
object using Equations (3), (4) and (6). 

[0017] 

However, when pitching of the vehicle takes place often, the vision axis 0 of camera (I ) 
changes according to the pitching angle, and the magnitude of the change is unclear. 
Consequently, in this case, it is impossible to compute the y-coordinate of the object, width Ws 
of the object, ajid distance D from the focal position of camera (1) to the object using the 
aforementioned processing. Consequently, for an image for which image acceleration (3c) is not 
zero, these can be computed as follows. 

[0018] 

First of all, the balanced images, captured at different times tl and T2, of the presence of 
the object judged to be the same object in the image for which image acceleration (3c) is not zero, 
are read from image memory (2). Whether the objects detected in the ijiiages with a non-zero 
image acceleration (3c) are the same object can be judged by checking whether they have a 
similar velocity in the images and a similar shape after edge extraction processing and detection. 
Also, camera (1 ) in this embodiment is a high-speed camera, and it takes consecutive pictures 
from the front of the vehicle at a minute prescribed time interval At, such as 2 nis. The 
precondition is that the balanced images captured at different times Tl and T2 must contain the 
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same object detected in the image with non-zero image acceleration (3c). The following 
explanation is given based on this precondition. 

[0019] 

It is possible to represent tlie distances Dl and D2, between Ihe focal position of camera 
(1) and the object in the images balanced at times Tl and T2, with the following Equations (7) 
and (8) using Equation (4) by utilizing the apparent angles 00 1 and 0o2 of the object at said 
times, respectively. 

DI^H/taniO-f-Ool)--- (7) 
02 = H / t an {Q-^BoZ )• • • ( 8 ) 

[0020] 

Also^ widths Ws of the object at times Tl and 12 can be represented with the following 
Equations (9) and (10) using Equation (6) from the lateral angle 6x1 and 9x2 of the object at said 
times, respectively. 

Ws =Gx V D 1 — M 9 ) 
Ws =0x 2- D 2— M 1 0 ) 

As a result, by substituting Equations (7) and (8) into Equations (9) and (10), respectively, 
one can obtain the following Equations (11) and (12): • 

Ws=ex1- H/tan(u + eo1)'' • (11) 
Ws=Gx2' H / tan ( 0 + Go 2 ) ■ • • (12) 

[0021] 

In Equations (11) and ( 1 2), when an image from far the front of the vehicle is captured 
with on-board camera (1 ), it is possible to set approximately G ~ 0, Go 1 - 0, and Go2 ~ 0. 
Consequently, in Equation (11), one has tan (0+Gol) G + Ool . In Equation (12), one has tan 
(0+Go2) -> G + Go2. As a result. Equations (1 1) and (12) are represented by (bllowing Equations 
(13) and (14), respectively. Baised on this relationship, one can obtain the following Equations 
(15) and (16): 

Ws =ex 1 • H / ( e+eo 1 ) • • * { i 3 ) 
Ws =ex 2 • H / ( e+eo 2 ) • • • ( 1 4 ) 

Ws- (O+Oo 1) =0x 1 ' H- - • (15) 
WS' (f) + 0o 2 ) =0x 2- H- • • (16) 
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[0022] 

Here, by subtj-acting Equation (16) from Equation (15), the following Equation (17) can 
be obtained, and it is possible to compute width Ws of the object. 

Ws = H- (9x1-ex2) / (eo1'.eo2)-'- (17) 

As a result, even when pitching of the vehicle takes place, and the vision axis 0 of camera 
(1) for the images captured in this case becomes unknown, it is still possible to compute width 
Ws of the object from the camera mounting height H that can be detected i n the two images 
balanced at different times for the same ol:)ject, as well as from 00 K 002, Ox 1 and 0x2 that can be 
measured from the images. 

[0023] 

Based on the width Ws of the object computed in this case, distance D from the focal 
position of camera (1) to the object in the images captured during said pitching slate, it is 
possible to derive the following Equation (18) using Equation (6). 

D=Ws/9x =Ws-p/xw — (18) 

As a result, by means of Equation (4), vision axis 0 of camera (1) can be computed with 
the following Equation ( 1 9) using distance D from the focal position of camera (1 ) to the object. 

0= a t a n ( H / D ) -Go- • • (19) 

As a result, even when there is change in vision axis 0 of camera (1) when pitching of the 
vehicle takes place, it is still possible to compute width Ws of the object and vision axis 0 of 
camera (I) based on the images balanced [captured] at different times Tl and T2. Since vision 
axis 0 of camera (1) has been computed, it is possible to compute the y-coordinate of the object 
in the image using Equation (3). 

[0024] 

Based on the y-coordinate in the image computed using the aforementioned processing, 
for example, it is possible to mark the object on a bird's-eye-view map displayed on display (4). 
As a result, even when pitching of the vehicle takes place, it is still possible to reliably correct for 
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the deviation of the object in the y-direction in the image due to pitching, and to display the 
obtained resuh in a bird's-eye-view map. 

[0025] 

Figure 5 is a flow chart illustrating the process of object-detecting device (100) in this 
embodiment. The processing shown in Figure 5 is carried out as Ibllows: the ignition switch of 
the vehicle is turned ON, the power supply for the object-detecting device is turned ON. and the 
program is started that is used to execute the processing with microcomputer (3). hi step S 10, the 
picked~up images captured with camera (I) and stored hi image memory (2) are read, and 
process flow then continues to step S20. 

[0026] 

hi step S20, edge extraction processing is performed on any image that is read as 
described above to compute the optical flow. As a result, image velocity (3b) and image 
acceleration (3c) are computed, and process flow continues to step S30. In step S30, 
determination is made as to whether the computed image acceleration (3c) is zero. If it is 
determined that the computed image acceleration (3c) is zero, the read image was judged to have 
been captured when the vehicle was balanced, and process How continues to step S40, In step 
S40, as explained above, the y-coordinate of the object, width Ws of the object, and distance D 
from the focal position of camera (l) ,to the object are computed by means of said Equations (3), 
(4) and (6). 

[0027] 

On the other hand, when it is determined that the computed image acceleration (3c) is not 
zero, it is judged that the read image was captured when pitching was taking place, and process, 
flow continues to step S50. hi step S50, the images captured in the balanced state at different 
times Tl and T2 and containing the same object as said object detected in the picked-up image 
are read from image memory (2). Process flow then continues to step S60. Width Ws of the 
object in real space, distance D from the focal position of camera (1) to the object, and vision 
axis 0 of camera (1) are then computed by means of Equations (17), (18), and (19), Then, process 
flow continues to step 870, and the y-coordinate of the object is computed using Equation (3). 

[0028] 

Then, process flow continues to step S80. In this step, based on the y-coordinate of the 
ol)ject and width Ws of the object in real space, the detected object is mapped on a bird's-eye- 
view map, and this is displayed on display unit (4), Process flow then continues to S90. In step 
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S90, a judgjnent is made as to whether the ignition switch of the vehicle is OFF, If it is not OFF. 
flow returns to step SIO and the aforementioned process is repeated. If it is OFF, the processing 
comes to an end. 

[0029] 

In the present embodiment explained above, the following effects can be realized. 

(1) Said image acceleration (3c) of the image of the object is computed, and the image for 
which image acceleration (3c) for the image of tlie object is found not to be zero is judged to be 
an image captured when the vehicle was pitching. As a result, it is possible to detect the 
occuiTcnce of pitching without carrying a device for detecting the posture of the vehicle or 
another device for detecting pitching, so that the cost of the device can be reduced with this 
constitution. 

(2) For the image captured when no pitching of the vehicle takes place, that is, for an 
image captured in the balanced state, distance D from the focal position of camera (1) to the 
object is computed based on camera mounting height H, vision axis 0 of camera (i), and 
apparent angle of the object 0o, and object width Ws can be computed based on image width xw, 
object lateral angle 0x, and the distance from the focal position of camera (1) to the object. As a 
result, the distance to the object and the size of the object can be detected without any need for a 
dedicated sensor, and the device can be realized with a simple constitution. 

[0030] 

(3) When pitching of the vehicle takes place, width Ws of the object in real space is 
computed based on the images captured in the balanced slate at different times Tl and T2 
determining that the object is the same as that detected in the image when pitching took place, 
and distance D from the focal position of camera (1) to the object and vision axis 9 of camera (1 ) 
are computed based on the computing result. As a result, even when vision axis 0 of camera (1 ) 
is uncertain when pitching takes place, it is still possible to correctly compute the vision axis 0 of 
camera (1), the distance to the detected object, and the size of the detected object. 

[0031] 

(4) Also, when pitching of the vehicle takes place, the y-coorclinate of the object in the 
image is computed based on vision axis 0 of camera (1) and (he object is mapped on the bird's- 
eye-view map and displayed on display unit (4). As a result, even wheji pitching of the vehicle 
takes place, it is still possible to reliably correct for deviation iji the y-direction in the image of 
the obj.ect due to pitching, and to display the collected result on the bird's-eye-view map. 
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[0032] 

Also, the following modifications are allowed. 

(.1) In the aforementioned embodiment, in order to detect image velocity (3b) and image 
acceleration (3c), edge extiaction processing is performed on the picked-up image, and the 
optical flow is computed. However, other schemes can be adopted to detect image velocity (3b) 
and image acceleration (3c). 

[0033] 

(2) hi the aforementioned embodiment, the image for which iinage acceleration (3c) is 
zero is judged to be an image captured when the vehicle itself was balanced. However, it is also 
possible to judge that an image for which image velocity (3b) is plus or minus and image 
acceleration (3c) is zero is an iniage captured when the vehicle itself was balanced. In this way, 
eveji when the characteristics on the extension side and those on the contraction side are different 
due to the vehicle suspension, it is still possible to correctly detect the balance state of the vehicle. 

[0034J 

(3) In said embodiment, as an example, it is assumed that there is only one moving object 
in the image captured with camera (1). Hov/ever, the present invention is not limited to this 
scheme. For example, a scheme can be adopted in which plural moving objects are present in the 
image. In this case, said processing is performed for all of the objects detected in the image, and 
the y-coordinate of an object, width Ws of the object in real space, and distance D from the focal 
position of camera (1) to the object are then computed. 

[0035] 

(4) In said embodiment, as an example, when pitching of the vehicle takes place, vision 
axis 0 of camera (I) becomes unclear, so that width Ws of the object in real space is computed, 
and distance D from the focal position of camera (1) to the object, vision axis 0 of camera (1), 
and the y-coordinate of the olyect in the image are computed. However, the present invention is 
not limited to this scheme. For example, a scheme can also be adopted in which even v^hen 
mounting position of camera (I) deviates and vision axis 0 of camera (1) becomes unclear, the 
aforementioned method is used to compute width Ws of the object in real space, distance D from 
the focal posi tion of camera (1 ) to the object; vision axis 0 of camera (1), and the y-coordinate of 
the object in the image. 
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[0036] 

(5) In said embodiinent, as an example, the delected object is mapped on a bird's-eye- 
view map for display on display unit (4). However, the present invention is not limited to this 
scheme. For example, a scheme can be adopted in which the object is mapped on a planar map or 
on another type of map for display. 

[0037] 

In the following, an explanation will be given regarding the correlations between the 
structural elements of the embodiments and the claims. Said camera ( 1 ) corresponds to the image 
pickup means, and microcomputer (3) corresponds to the image judgment means, object position 
computing means, acceleration computing means, velocity computing means, and object 
information computing nieans. Also, as long as the characteristic features of the present 
invention are observed, the present invention is not limited to the aforementioned embodiments. 

Brief description of the Figures 
[0038] 

Figure 1 is a block diagram illustrating an example of tlie constitution of an embodiment 
of the object-detecting device. 

Figure 2 is a diagram ilhistrating an example of change in the position of the object 
present the front of the vehicle and present in the picked-up image when pitchiiig of the vehicle 
takes place. 

Figure 3 is a graph illustrating the relationship between pitching angle 0p, the velocity in 
the y-direction of the object in the image, and the acceleration in the y-direction of the object in 
the image. 

Figure 4 is a diagram illustrating an example of the situation when the images that have 
been captured balanced are used to compute the height and width of the object in real space, and 
the distance from the focal position of camera (1) to the object. 

Figure 5 is a flow chart illustrating the processing performed by object-detecting device 

(100). 

Explanation of symbols 
[0039] 

1 00 Object-delecting device 

1 Camera 

2 Image jnemory 

3 Microcomputer 



